-
Notifications
You must be signed in to change notification settings - Fork 722
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added Configurable Delay option #268
base: master
Are you sure you want to change the base?
Conversation
This helps avoid the rate-limiting introduced by archive.org
I was unable to run rake on this due to an error with rake. I don't think i broke any of the tests but let me know if you need anything fixed. |
I picked -n for the delay option short form based on the linux nice command. the other good options were already used. Feel free to change this if desired. |
@sww1235 it would be nice to update |
I think it would be nice to include a message on "connection refused" too: |
Note: you should add the download delay after the check if the file exists (this line): Otherwise you will be waiting a long time if you are resuming a partial download. There should also be a configurable amount of retry attempts per file. |
Great fork, has completely solved the issues I was having before. Thank you. |
@sww1235 you are referring to the With #267 (comment) it might be that this work-around is no longer essential, but it would be good to have either way. As for the parameter naming, what about |
Hello, I'm not a coder just a regular guy that ran into this problem. Is there anywhere I can read or watch on how to fix it? I see your fixes but how do I do it? Thanks |
The delay actually not the delay in between downloads/requests, but in between files processing, |
This helps avoid the rate-limiting introduced by archive.org
Alternative to #266 that allows for a configurable delay rather than a hardcoded delay. it also introduces the same delay when fetching snapshots
Closes #267, maybe #264, #244 and #246
Reference to archive.org implementing rate limiting: https://archive.org/details/toomanyrequests_20191110